TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode by Aggarwal-Raghav · Pull Request #458 · apache/tez

Aggarwal-Raghav · 2026-01-27T18:26:48Z

Please check screenshots in TEZ-4685

tez-yetus · 2026-01-27T18:47:37Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ master Compile Tests _
+1 💚	mvninstall	8m 56s		master passed
+1 💚	compile	0m 24s		master passed
+1 💚	checkstyle	0m 51s		master passed
+1 💚	javadoc	0m 24s		master passed
-1 ❌	spotbugs	1m 3s	/branch-spotbugs-tez-dag.txt	tez-dag in master failed.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	codespell	0m 27s		No new issues.
+1 💚	compile	0m 18s		the patch passed
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 11s		the patch passed
+1 💚	javadoc	0m 10s		the patch passed
-1 ❌	spotbugs	0m 17s	/patch-spotbugs-tez-dag.txt	tez-dag in the patch failed.
			_ Other Tests _
+1 💚	unit	5m 25s		tez-dag in the patch passed.
+1 💚	asflicense	0m 11s		The patch does not generate ASF License warnings.
		20m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/1/artifact/out/Dockerfile
GITHUB PR	#458
Optional Tests	dupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile
uname	Linux cdb2f8f4b9fb 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-458/src/.yetus/personality.sh
git revision	master / `f2eba56`
Default Java	Ubuntu-21.0.9+10-Ubuntu-124.04
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/1/testReport/
Max. process+thread count	246 (vs. ulimit of 5500)
modules	C: tez-dag U: tez-dag
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/1/console
versions	git=2.43.0 maven=3.8.7 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

…zookeeper mode

Aggarwal-Raghav · 2026-02-18T18:33:46Z

@ayushtkn / @abstractdog , I have updated the patch. Can you provide your feedback on this implementation?

tez-yetus · 2026-02-18T19:14:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	14m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ master Compile Tests _
+1 💚	mvninstall	11m 10s		master passed
+1 💚	compile	0m 45s		master passed
+1 💚	checkstyle	1m 5s		master passed
+1 💚	javadoc	0m 35s		master passed
+0 🆗	spotbugs	2m 28s		tez-dag in master has 749 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	codespell	0m 57s		No new issues.
+1 💚	compile	0m 32s		the patch passed
+1 💚	javac	0m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 18s		the patch passed
+1 💚	javadoc	0m 13s		the patch passed
+1 💚	spotbugs	1m 49s		the patch passed
			_ Other Tests _
+1 💚	unit	5m 33s		tez-dag in the patch passed.
+1 💚	asflicense	0m 12s		The patch does not generate ASF License warnings.
		41m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/2/artifact/out/Dockerfile
GITHUB PR	#458
Optional Tests	dupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile
uname	Linux 2fa494d80081 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-agent/workspace/tez-multibranch_PR-458/src/.yetus/personality.sh
git revision	master / `80f79e8`
Default Java	Ubuntu-21.0.10+7-Ubuntu-124.04
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/2/testReport/
Max. process+thread count	238 (vs. ulimit of 5500)
modules	C: tez-dag U: tez-dag
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-458/2/console
versions	git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2026-02-20T11:00:24Z

tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java

-      Configuration conf = new Configuration();
+      Configuration conf =
+          STANDALONE_ZOOKEEPER.name().equals(System.getenv(TezConstants.TEZ_FRAMEWORK_MODE))
+              ? new TezConfiguration()
+              : new Configuration();


This should be kept somehow in FrameworkUtils, like FrameworkUtils.getInitialConfiguration()
because we might want to hide the details of this "hack" from DAGAppMaster.

Why I call it a hack: it's not because the solution is bad, it's because of the current design, which leads to a catch-22:

we need a Configuration

for that we need to know the framework mode

to know the framework mode and get framework client/server classes we need a Configuration object: getFrameworkService(conf)

so we cannot do anything better than to achieve original Configuration by the value stored in environment...which is fine, but DAGAppMaster's already overwhelmed logic should not contain such a thing

@abstractdog do you have pointer what is different in STANDALONE_ZOOKEEPER that it doesn't work here but does work in YARN mode?

hm, that's a good question: I guess here is where we might want to get back to the original motivation: whether a standalone TezAM docker container needs a tez-site.xml on its classpath (and if so, could it read) or if there is another way to configure it... isn't TEZ-4014 or some earlier work already taking care of this? what do you think @Aggarwal-Raghav ?

tez/tez-common/src/main/java/org/apache/tez/common/TezUtilsInternal.java

Lines 100 to 105 in f09ba7f

for (String confFile : confFromXml.getTrimmedStringCollection(TezConfiguration.TEZ_AM_STANDALONE_CONFS)) {

try (InputStream additionalInput = ClassLoader.getSystemResourceAsStream(confFile)) {

Configuration additionalConfFromXml = TezUtilsInternal.readTezConfigurationXml(additionalInput);

confFromXml.addResource(additionalConfFromXml);

}

}

I'll get back on this TEZ-4014 analysis in couple of days.

Thanks, @abstractdog, for pointing out TEZ-4014.It works well, provided we pass TEZ_FRAMEWORK_MODE=STANDALONE_ZOOKEEPER as an environment variable (using the --env-file or -e flag in the docker run command) and ensure tez-site.xml is in the classpath.

Without this environment variable, it defaults back to YARN mode. I might have missed passing the -e flag initially while working on TEZ-4682 stared using with new TezConfiguration() as in current TEZ-4682 PR 551075b and never checked back :-(

I'll close this JIRA then

Aggarwal-Raghav mentioned this pull request Jan 27, 2026

TEZ-4682: [Cloud] Tez AM docker image #456

Open

TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in …

80f79e8

…zookeeper mode

Aggarwal-Raghav force-pushed the TEZ-4685 branch from f2eba56 to 80f79e8 Compare February 18, 2026 18:32

Aggarwal-Raghav changed the title ~~TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath~~ TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode Feb 18, 2026

abstractdog reviewed Feb 20, 2026

View reviewed changes

Aggarwal-Raghav closed this Feb 21, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode#458

TEZ-4685: DagAppMaster is not picking tez-site.xml from classpath in zookeeper mode#458
Aggarwal-Raghav wants to merge 1 commit intoapache:masterfrom
Aggarwal-Raghav:TEZ-4685

Aggarwal-Raghav commented Jan 27, 2026

Uh oh!

tez-yetus commented Jan 27, 2026

Uh oh!

Aggarwal-Raghav commented Feb 18, 2026

Uh oh!

tez-yetus commented Feb 18, 2026

Uh oh!

abstractdog Feb 20, 2026 •

edited

Loading

Uh oh!

ayushtkn Feb 20, 2026

Uh oh!

abstractdog Feb 20, 2026 •

edited

Loading

Uh oh!

Aggarwal-Raghav Feb 20, 2026

Uh oh!

Aggarwal-Raghav Feb 21, 2026

Uh oh!

Aggarwal-Raghav Feb 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	for (String confFile : confFromXml.getTrimmedStringCollection(TezConfiguration.TEZ_AM_STANDALONE_CONFS)) {
	try (InputStream additionalInput = ClassLoader.getSystemResourceAsStream(confFile)) {
	Configuration additionalConfFromXml = TezUtilsInternal.readTezConfigurationXml(additionalInput);
	confFromXml.addResource(additionalConfFromXml);
	}
	}

Comments

Conversation

Aggarwal-Raghav commented Jan 27, 2026

Uh oh!

tez-yetus commented Jan 27, 2026

Uh oh!

Aggarwal-Raghav commented Feb 18, 2026

Uh oh!

tez-yetus commented Feb 18, 2026

Uh oh!

abstractdog Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ayushtkn Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

abstractdog Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Feb 21, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Feb 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abstractdog Feb 20, 2026 •

edited

Loading

abstractdog Feb 20, 2026 •

edited

Loading